Deep learning technology has made great achievements in the field of image. In order to defend against malware attacks, researchers have proposed many Windows malware detection models based on deep learning. However, deep learning models are vulnerable to adversarial example attacks. Malware can generate adversarial malware with the same malicious function to attack the malware detection model and evade detection of the model. Currently, many adversarial defense studies have been proposed, but existing adversarial defense studies are based on image sample and cannot be directly applied to malware sample. Therefore, this paper proposes an adversarial malware defense method based on adversarial training. This method uses preprocessing to defend simple adversarial examples to reduce the difficulty of adversarial training. Moreover, this method improves the adversarial defense capability of the model through adversarial training. We experimented with three attack methods in two sets of datasets, and the results show that the method in this paper can improve the adversarial defense capability of the model without reducing the accuracy of the model.

Malware detection models based on deep learning have been widely used, but recent research shows that deep learning models are vulnerable to adversarial attacks. Adversarial attacks are to deceive the deep learning model by generating adversarial samples. When adversarial attacks are performed on the malware detection model, the attacker will generate adversarial malware with the same malicious functions as the malware, and make the detection model classify it as benign software. Studying adversarial malware generation can help model designers improve the robustness of malware detection models. At present, in the work on adversarial malware generation for byte-to-image malware detection models, there are mainly problems such as large amount of injection perturbation and low generation efficiency. Therefore, this paper proposes FGAM (Fast Generate Adversarial Malware), a method for fast generating adversarial malware, which iterates perturbed bytes according to the gradient sign to enhance adversarial capability of the perturbed bytes until the adversarial malware is successfully generated. It is experimentally verified that the success rate of the adversarial malware deception model generated by FGAM is increased by about 84\% compared with existing methods.

The current state-of-the-art Android malware detection systems are based on machine learning and deep learning models. Despite having superior performance, these models are susceptible to adversarial attacks. Therefore in this paper, we developed eight Android malware detection models based on machine learning and deep neural network and investigated their robustness against adversarial attacks. For this purpose, we created new variants of malware using Reinforcement Learning, which will be misclassified as benign by the existing Android malware detection models. We propose two novel attack strategies, namely single policy attack and multiple policy attack using reinforcement learning for white-box and grey-box scenario respectively. Putting ourselves in the adversary's shoes, we designed adversarial attacks on the detection models with the goal of maximizing fooling rate, while making minimum modifications to the Android application and ensuring that the app's functionality and behavior do not change. We achieved an average fooling rate of 44.21% and 53.20% across all the eight detection models with a maximum of five modifications using a single policy attack and multiple policy attack, respectively. The highest fooling rate of 86.09% with five changes was attained against the decision tree-based model using the multiple policy approach. Finally, we propose an adversarial defense strategy that reduces the average fooling rate by threefold to 15.22% against a single policy attack, thereby increasing the robustness of the detection models i.e. the proposed model can effectively detect variants (metamorphic) of malware. The experimental analysis shows that our proposed Android malware detection system using reinforcement learning is more robust against adversarial attacks.

To address the costs of reverse engineering and signature extraction, advanced research on malware detection focuses on using neural networks to learn malicious behaviors with static and dynamic features. However, deep learning-based malware detection models are vulnerable to a hack from adversarial samples. The attackers' goal is to generate imperceptible perturbations to the original samples and evade detection. In the context of malware, the generated samples should have one more important character: it should not change the malicious behaviors of the original code. So the original features can not be removed and changed. In this paper, we proposed a reinforcement learning based attack to deceive graph based malware detection models. Inspired by obfuscation techniques, the central idea of the proposed attack is to sequentially inject semantic Nops, which will not change the program's functionality, into CFGs(Control Flow Graph). Specifically, the Semantics-preserving Reinforcement Learning(SRL) Attack is to learn an RL agent to iteratively select the semantic Nops and insert them into basic blocks of the CFGs. Variants of obfuscation methods, hill-climbing methods, and gradient based algorithms are proposed: 1) Semantics-preserving Random Insertion(SRI) Attack: randomly inserting semantic Nops into basic blocks.; 2) Semantics-preserving Accumulated Insertion(SAI) Attack: declining certain random transformation according to the probability of the target class; 3) Semantics-preserving Gradient based Insertion(SGI) Attack: applying transformation on the original CFG in the direction of the gradient. We use real-world Windows programs to show that a family of Graph Neural Network models are vulnerable to these attacks. The best evasion rate of the benchmark attacks are 97% on the basic GCN model and 96% on DGCNN model. The SRL attack can achieve 100% on both models.